Induced Weights Artificial Neural Network

It is widely believed in the pattern recognition field that the number of examples needed to achieve an acceptable level of generalization ability depends on the number of independent parameters needed to specify the network configuration. The paper presents a neural network for classification of high-dimensional patterns. The network architecture proposed here uses a layer which extracts the global features of patterns. The layer contains neurons whose weights are induced by a neural subnetwork. The method reduces the number of independent parameters describing the layer to the parameters describing the inducing subnetwork. Introduction The great potential of the neural networks is most frequently used in pattern recognition. The most challenging problem here is achieving the proper generalization. Typical images and time-series are usually large, often with several hundred variables. Fully connected, unrestricted networks do not work well as far as recognizing such large patterns is concerned. The number of examples needed to achieve an acceptable level of generalization ability is dependent on the intrinsic entropy of the chosen architecture, and can be decreased by reducing the number of independent parameters needed to specify the network configuration. One of the ways to improve generalization is a reduction of the network structure on the base of a pruning algorithm (e.g. the Optimal Brain Damage [7]). Another deficiency of the fully-connected architectures is that the topology of the inputs is entirely ignored. In fact, images have a strong 2D structure, while time-series have a strong 1D structure. Pixels, or variables, spatially or temporally adjacent are correlated. The application of a specialized network architecture, instead of a fully-connected net, can reduce the number of free parameters. There are many papers that propose specialized network architectures for the recognition of large patterns. Convolutional networks, for instance, use the techniques of local receptive fields and shared weights. These networks extract and combine local features of pattern [4,5]. Principal component analysis transforms a number of correlated variables into a smaller number of uncorrelated variables called principal components and is frequently adopted for dimensionality reduction [1]. The idea that has been followed in the IWANN is based on the invention of dynamically-calculated weights, which results in giving the individual neurons the ability to transform large patterns, using only a limited number of parameters describing their connection weights. The proposed network architecture extracts and transforms the global features of the patterns. Network Architecture The IWANN network makes use of dynamically-calculated connection weights. As a result, the number of parameters describing neural network connections is reduced. Fig. 1. Induced weights network scheme The input layer of the network proposed can be oneor multidimensional, and every neuron in this layer is described by its geometric position. The layer is a data source for the induced weights layer. It contains radial basis neurons which apply the Gaussian transformation function. The input of this radial basis transformation function is the Euclidean distance between the input vector and the vector of weights calculated by the inducing subnetwork, multiplied by the bias. ( ) ( ) ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − = ∑ = ) 0 ( 1 2 0 ) ( ) 1 ( ) 1 ( 2 1 N j j M ji i i y y b y & & φ (1) where, ) 1 ( i y output of the i-th neuron in induced weights layer, () φ Gaussian transformation function, ) 1 ( i b bias of neuron, ) (M ji y & & output of neuron in output (M-th) layer of inducing network equal to the weight of the j-th input of the i-th neuron of the induced layer, and ( ) o j y output of j-th input neuron – network input The task of the inducing neural subnetwork consists in positioning of high-dimensional centers. The inducing network is a multilayer perceptron: ( ) ( ) ( ) ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ = ∑ −